Learning from multiple annotators: Distinguishing good from random labelers
نویسندگان
چکیده
0167-8655/$ see front matter 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.patrec.2013.05.012 ⇑ Corresponding author. Tel.: +351 239790056. E-mail addresses: [email protected] (F. Rodrigues), [email protected] (F. Pereira), [email protected] (B. Ribeiro). 1 Tel.: +65 93233653. 2 Tel.: +351 239790056. 3 http://www.mturk.com. Filipe Rodrigues a,⇑, Francisco Pereira , Bernardete Ribeiro a,2
منابع مشابه
Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations
Supervised machine learning methods to model word sense often rely on human labelers to provide a single, ground truth sense label for each word in its context. The finegrained, sense label inventories preferred by lexicographers have been argued to lead to lower annotation reliability in measures of agreement among two or three human labelers (annotators). We hypothesize that annotators can ag...
متن کاملA Probabilistic Framework to Learn from Multiple Annotators with Time-Varying Accuracy
This paper addresses the challenging problem of learning from multiple annotators whose labeling accuracy (reliability) differs and varies over time. We propose a framework based on Sequential Bayesian Estimation to learn the expected accuracy at each time step while simultaneously deciding which annotators to query for a label in an incremental learning framework. We develop a variant of the p...
متن کاملActive Learning for Crowdsourcing Using Knowledge Transfer
This paper studies the active learning problem in crowdsourcing settings, where multiple imperfect annotators with varying levels of expertise are available for labeling the data in a given task. Annotations collected from these labelers may be noisy and unreliable, and the quality of labeled data needs to be maintained for data mining tasks. Previous solutions have attempted to estimate indivi...
متن کاملGot Many Labels?: Deriving Topic Labels from Multiple Sources for Social Media Posts using Crowdsourcing and Ensemble Learning
Online search and item recommendation systems are often based on being able to correctly label items with topical keywords. Typically, topical labelers analyze the main text associated with the item, but social media posts are often multimedia in nature and contain contents beyond the main text. Topic labeling for social media posts is therefore an important open problem for supporting effectiv...
متن کاملA Classification Model for Diverse and Noisy Labelers
With the popularity of the Internet and crowdsourcing, it becomes easier to obtain labeled data for specific problems. Therefore, learning from data labeled by multiple annotators has become a common scenario these days. Since annotators have different expertise, labels acquired from them might not be perfectly accurate. This paper derives an optimization framework to solve this task through es...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- Pattern Recognition Letters
دوره 34 شماره
صفحات -
تاریخ انتشار 2013